Edit on GitHub

Realtime Audio Processing

This example demonstrates how to stream audio and apply real-time signal processing to the signal.

Real-time processing consists of a source, zero or more modifiers, and a sink. Sources generate the raw signal. Modifiers alter the signal. Sinks are a destination for the signals, typically a sound card, but in this example we use a buffer.

First the required packages are loaded and the sample rate and number of audio channels is specified.

using AuditoryStimuli, Unitful, Plots, Pipe, DSP

sample_rate = 48000
audio_channels = 2;
source_rms = 0.2

Set up the signal pipeline components

First a sink is generated. This would typically be a sound card, but that is not possible on a web site. Instead, for this website example a dummy sink is used, which simply saves the sample to a buffer.

sink = DummySampleSink(Float64, sample_rate, audio_channels)

# But on a real system you would use something like
# a = PortAudio.devices()
# sink = PortAudioStream(a[3], 0, 2)

We also need a source. Here we use a simple white noise source.

source = NoiseSource(Float64, sample_rate, audio_channels, source_rms)

And we will apply one signal modifier. The first modifier adjusts the amplitude of the signal. We want the signal to ramp from silent to full intensity, so we set the initial value to 0.0 and the target value to 1.0, and the maximum change per frame to 0.01.

amp = Amplification(1.0, 0.0, 0.05)

Run the real-time audio pipeline

Audio is typically processed in small chunks of samples called frames. Here we request a frame from the noise source with length 1/100th of a second, or 480 samples. This is then passed through the signal amplifier, then sent to the sink.

for frame = 1:100
    @pipe read(source, 0.01u"s") |> modify(amp, _) |> write(sink, _)
end

Verify processing was correctly applied

plot(sink.buf)

Apply a filter modifier

A filter can also be applied to the data as a modifier. The filter also maintains its state, so can be used in real time processing. Below a bandpass filter is designed, for more details on filter design using the DSP package see: https://docs.juliadsp.org/stable/filters/

responsetype = Bandpass(500, 4000; fs=48000)
designmethod = Butterworth(4)
zpg = digitalfilter(responsetype, designmethod)

Once the filter is specified as a zero pole gain representation two filters are instansiated using this specification. A filter must be generated for each channel of audio. These DSP.Filters are then passed in to the AuditoryStimuli filter object for further use.

f_left = DSP.Filters.DF2TFilter(zpg)
f_right = DSP.Filters.DF2TFilter(zpg)
bandpass = AuditoryStimuli.Filter([f_left, f_right])

Once the filters are designed and placed in an AuditoryStimule.Filter object they can be used just like any other modifier. Below the filer is applied to 1 second of audio in 1/100th second frames.

for frame = 1:100
    @pipe read(source, 0.01u"s") |> modify(amp, _) |> modify(bandpass, _) |> write(sink, _)
end

Modifying modifier parameters

The parameters of modifiers can be varied at any time. Below the target amplification is set to zero to ramp off the signal.

 setproperty!(amp, :target_amplification, 0.0)
for frame = 1:20
    @pipe read(source, 0.01u"s") |> modify(amp, _) |> modify(bandpass, _) |> write(sink, _)
end

Verify output

The entire signal (both the amplification, then the filtering) can be viewed using the convenience plotting function below. We observe that the signal is ramped on due to the amplification modifier. We can then see that at 1 second the spectral content of the signal was modified. And finally the signal is ramped off.

PlotSpectroTemporal(sink, figure_size=(800, 400), frequency_limits = [0, 8000])

Other tips

This example demonstrates the basics of real-time signal processing with this package. For a real application the following considerations may be required:

  • Running the audio stream in its own thread so you can process user input or run other code in parallel. This is easily accomplised using @spawn, see: example
  • Enable or disable processing rather than modifying the pipeline. Each modifier has an enable flag so that it can be disabled, when disabled the signal is simply passed through and not modified.